Skip to content

[test] try to find out why GPUs are not being utilized correctly #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 144 commits into
base: main
Choose a base branch
from
Open

[test] try to find out why GPUs are not being utilized correctly #27

wants to merge 144 commits into from

Conversation

MichaelHudgins
Copy link

No description provided.

nitins17 added 30 commits August 27, 2024 15:32
@nitins17
Add scripts, configs, and envs for the JAX's new CI
2634664
@nitins17
add workflow for running bazel gpu tests locally
738a7bf
@nitins17
make some tweaks
e901247
@nitins17
Fix runner naming
cab4ae3
@nitins17
check script name with regex match operator
f169816
@nitins17
clean up
0a5e1af
@nitins17
fix typos and clean up
8210c6a
@nitins17
fix path to env
6cfb12b
@nitins17
fix path to env (actually)
3aaa330
@nitins17
move wait for connection step to just before running tests
f2fb445
@nitins17
change docker image for bazel local gpu test workflow
222542d
@nitins17
change image to tag instead of sha
8fe81fb
@nitins17
set default Python version for all builds
6666472
@nitins17
set default Python version for jax-cuda-pjrt and read CLI argument fo…
f7b47cf 
…r others
@nitins17
Create a matrix for building JAX artifacts
057b8e3
@nitins17
add default python to matrix for jax-cuda-pjrt
843afe3
@nitins17
exlude pjrt from the python matrix
82f0454
@nitins17
add workflow and scripts for running pytests
23fb8c9
@nitins17
make pytest script executable
875a271
@nitins17
add workflow for pytest
6384f73
@nitins17
change workflow name
320bf83
@nitins17
try fixing workflow
204c029
@nitins17
fix runner name in pytest workflow
cc4e210
@nitins17
add connection wait for pytest workflow and fix typo in container fie…
4474934 
…ld of build artifacts workflow
@nitins17
dummy change; push to make github actions detect labels
6ae0257
@nitins17
dummy change; push to make github actions detect labels
d7ab9e8
@nitins17
set clang path for all linux ci builds
4858b38
@nitins17
install wheels with system python only for pytests
30a3951
@nitins17
Merge branch 'main' of https://github.com/google-ml-infra/jax-fork in…
ff2e5fe 
…to srnitin/task-jax-ci-rework
@nitins17
install missing docker dependencies
60f1231
@nitins17
fix installing wheels correctly
637b0a5
@nitins17
fix issue when installing wheels
460cb11
@nitins17
modify scripts to run tpu tests
c65c2b2
@nitins17
set jaxci python env var
c448ea6
@nitins17
set the tpu cores in the env file
5b450af
@nitins17
enable bazel gpu rbe tests
1418ba8
@nitins17
make github detect label change
9f739ab
@nitins17
change to python 3.10
391a913
@nitins17
install the right version of python deps for pytest cpu
3b98f9b
@nitins17
fix syntax error
dd41e8d
@nitins17
Debug GPU failures
ed31404
@nitins17
debug gpu vms getting killed
04537e7
@nitins17
debug gpu vms getting killed (2)
b880ed5
@nitins17
set cuda compute arg correctly
a8716ce
@nitins17
set cuda compute arg correctly (2)
de602b5
@nitins17
Merge branch 'main' of https://github.com/google-ml-infra/jax-fork in…
3783c84 
…to srnitin/task-jax-ci-rework
@nitins17
do not override xla repository when running gpu tests locally
bc6d9cb
@nitins17
bump device memory limit when running tests locally on single gpu
b8978b5
@nitins17
change bazel build to bazel run
26b1f59
@nitins17
increase test timeout
7b63e0c
@nitins17
update runner for bazel local gpu workflow
a7663fa
@MichaelHudgins
Change config back to stable driver
e25bbe8
@MichaelHudgins
Update build.py
e11b465 
The PJRT was being built for the wrong python
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MichaelHudgins @nitins17